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COLOR INTERPOLATION 
Background 

The invention relates to color interpolation. 

Fig. 1 shows a semiconductor imager 10 (e.g., a 
complementary metal-oxide semiconductor (CMOS) imager) might 
be used to electrically capture "snapshots" of an optical 
image. The imager is used to convert an optical image into 
an electrical representation. The imager 10 accomplishes 
this conversion through the use of an array of sensing 
elements arranged as pixel cells 12 that sense the intensity 
of light coming from the image. The "exposure time" for 
each snapshot depends on an integration interval during 
which each pixel cell 12 integrates an indication of the 
number of photons of light striking the cell 12 (i.e., 
measures an intensity of light striking the cell 12) and 
provides an indication of the integrated value via an analog 
output signal. For CMOS imagers, on-chip analog 
conditioning circuitry 14 (e.g w circuitry to perform 
correlated double sampling and gain control) and an analog- 
to-digital converter (ADC) 16 process the analog outputs of 
the pixel cells 12 to provide a digital representation of 



the image which can be retrieved from the imager 10 through 

a parallel port interface 18. 

The pixel cells 12 provide an indication of the * 

intensity of light striking the cell 12. Hence, the above- 

5 described arrangement may be used to produce a monochrome or 

luminance only representation of the image. However, to 
♦ 

produce color representations of the image, the imager also 
needs to provide information about primary colors (e.g., 
red, green and blue colors) of the image. To accomplish 

10 this, each pixel cell 12 is configured to sense the 

intensity level of light in one of the primary color bands. 
A typical way to accomplish this is to cover each pixel cell 
12 with a spectrum-discriminating filter (e.g., a filter 
that only allows a red, green or blue color band to pass 

15 through the filter) . As a result, some pixel cells 12 sense 
red light, some pixel cells 12 sense green light and some 
pixel cells 12 sense blue light. As an example, a multi- 
band filter pattern 20 (see Fig. 2) placed over the array of 
pixel cells 12 may have alternating red, green and blue 

20 filter stripes that extend along the columns of the array. 

Thus, each filter stripe of the pattern 20 configures one of 
the columns of the array to sense light in one of the 
primary color bands. As another example, the filter pattern 
may be checkered, instead of striped. 
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Each pixel cell 12 captures a portion of the image. 
To maximize the resolution of the image when reproduced on a 
display, it is desirable to form a one-to-one correspondence 
between the pixel cells 12 of the imager 10 and pixels of 
5 the display. However, with color imagers, three adjacent' 
pixel cells 12 (each pixel cell 12 sensing a different 
primary color band) are typically used to provide the 
information needed to form one pixel on the display. Thus, 
when used to capture color images, the effective display 

10 pixel resolution of the imager 10 typically is one third of 
the actual pixel cell 12 resolution. 

For purposes of preserving a one-to-one 
correspondence between the pixel cells 12 and the pixels of 
the display, one solution is to form an imager having three 

15 times as many pixel cells as corresponding pixels of the 
display to compensate for the three primary colors. 
Referring to Fig. 3, another solution is to use three 
imagers 22, 24, and 28, one for each primary color band of 
the image. Thus, for example, one imager 22 (covered by a 

20 red filter) senses red light, one imager 24 (covered by a 

green filter) senses green light, and one imager 26 (covered 
by a blue filter) senses the blue light comimg from the 
image. Dichroic plates 28 may be used to split the light 
into beams into its primary colors. 



Referring to Fig. 4, a third solution might be to 
use an off chip discrete- time signal processing (DSP) engine 
30 to interpolate the two missing colors for each pixel cell 
12. To accomplish this, the DSP engine 30 processes the 
5 color information provided by adjacent pixel cells 12. 

Typically, nearest neighbors are weighted with predetermined 
coefficients and averaged to determine a color at a 
particular pixel cell location. For example, referring back 
to Fig. 1, a pixel cell 12a that is covered by a red filter 

10 provides a representation of a red color of the portion of 
the image striking the cell 12a. To ascertain the blue 
color of the portion of the image otherwise striking the 
cell 12a (if not for the red filter) , the DSP engine 30 
averages (a weighted representation of) the outputs of 

15 adjacent pixel cells 12b and 12c (i.e., adjacent pixel cells 
covered by a blue filter) to interpolate the missing blue 
color. The DSP engine 3 0 also interpolates the green color 
of the portion of the image that would other strike the cell 
12a in a similar manner. 

20 Summary of the Invention 

In general, in one aspect, the invention features an 
imager that has first and second photosensitive sites and an 
interpolator located in a semiconductor substrate. The 
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first photosensitive site is configured to receive light 
having a spectral component, and the second photosensitive 
site is configured to measure the level of the spectral 
component in light received by the second photosensitive 
5 site. The interpolator is configured to estimate the level 
of the spectral component in the light received by the first 
photosensitive site based on the measurement by the second 
photosensitive site. 

Implementations of the invention may include one or 

10 more of the following. The first and/or second 

photosensitive sites may include a pixel cell and a filter 
that covers the pixel cell. The filter covering the first 
photosensitive site may be configured to prevent the 
spectral component from striking the pixel cell, and the 

15 filter covering the second photosensitive site may be 

configured to allow the spectral component to strike the 
pixel cell. The first photosensitive site may also be 
configured to measure the level of another spectral 
component in light received by the first photosensitive 

20 site, and the interpolator may be also configured to 

estimate the level of the another spectral component in the 
light received by the second photosensitive site based on 
the measurement by the first photosensitive site. 

The imager may also include a third photosensitive 

25 site (also located in the substrate) that is configured to 



measure the level of the other spectral component in light 
received by the third photosensitive site. The first 
photosensitive site may also be configured to receive light 
having the another spectral component, and the interpolator 
may also be configured to estimate the level of the spectral 
components in the light received by the first photosensitive 
site based on the measurements by the second and third 
photosensitive sites . 

In general, in another aspect, the invention 
features an imager that has first and second photosensitive 
sites and an interpolator located in a semiconductor 
substrate. Each first photosensitive site is configured to 
receive light having a spectral component, and each second 
photosensitive site is configured to measure the level of 
the spectral component in light received by the second 
photosensitive site. The interpolator is configured to 
estimate the level of the spectral component in the light 
received by at least one of the first photosensitive sites 
based on the measurements by the second photosensitive 
sites . 

Implementations of the invention may include one or 
more of the following. The interpolator may include an 
averaging circuit that is configured to perform the 
estimation by averaging some of the measurements by the 
second photosensitive sites. The interpolator may also 



* • 

include a scaling circuit that is configured to scale some 
of the measurements by predetermined coefficients before 
being averaged by the averaging circuit. The scaling 
circuit may be programmable to change one or more of the 
5 coefficients. The first and second photosensitive sites may 
be part of an array of photosensitive sites (e.g., located 
in a column of the array, a row of the array, or arranged in 
a rectangular block of an array) . 

In general, in another aspect, the invention 

10 features a color imager for use with light having first, 

second and third primary color bands. The imager has first, 
second and third photosensitive sites and an interpolator 
located in a semiconductor substrate. Each first 
photosensitive site is configured to receive a portion of 

15 the light and measure a level of the first primary color 
band in the portion of light received by the first 
photosensitive site. Each second photosensitive site is 
configured to receive a portion of the light and measure a 
level of the second primary color band in the portion of 

20 light received by the second photosensitive site. Each 

third photosensitive site is configured to receive a portion 
of the light and measure a level of the third primary color 
band in the portion of light received by the third 
photosensitive site. The interpolator is configured to 

25 estimate the levels of the second and third primary color 



bands in the light received by the first photosensitive 
sites based on the measurements by the second and third 
photosensitive sites; estimate the levels of the first and 
third primary color bands in the light received by the 
second photosensitive sites based on the measurements by the 
first and third photosensitive sites; and estimate the 
levels of the first and second primary color bands in i the 
light received by the third photosensitive sites based on 
the measurements by the first and second photosensitive 
sites . 

Implementations of the invention may include one or 
more of the following. The interpolator may be also 
configured to furnish a representation of the levels of the 
first, second and third primary color bands for each of the 
first, second and third photosensitive sites. The 
representation for each site may include a representation 
(e.g., a true color representation) of the color of the 
light received by the site. 

In general, in another aspect, the invention 
features a method that includes using a first photosensitive 
site located in a semiconductor substrate to receive light 
having a spectral component. A second photosensitive site 
located in the substrate is used to measure the level of the 
spectral component in light received by the second 
photosensitive site. An interpolator located in the 



substrate is used to estimate the level of the spectral 
component in the light received by the first photosensitive 
site based on the measurement by the second photosensitive 
site. 

5 In general, in another aspect, the invention 

features a method that includes using first photosensitive 
sites located in a semiconductor substrate to receive light 
having a spectral component. Second photosensitive sites 
located in the substrate are used to measure the level of 

10 the spectral component in light received by each of the 

second photosensitive sites. An interpolator located in the 
substrate is used to estimate the level of the spectral 
component in the light received by at least one of the first 
photosensitive sites based on the measurements by the second 

15 photosensitive sites. 

Among the advantages of the invention are one or 
more of the following. True color imaging occurs on a 
single semiconductor chip. The pixel cells of the imager 
and the pixels of the display have a one-to-one 

2 0 correspondence Only one imager is required. The imager 
may be used with many commonly used color filter patterns. 

Other advantages will become apparent from the 
following description and from the claims. 
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Fig. 1 is a schematic view of a semiconductor 

imager. 

Fig. 2 is a schematic view of color filters. 

Fig. 3 is a block diagram of a system to interpolate 

5 color. 

Fig. 4 . is a schematic view of an optical system to 
separate light into primary color components. 

Fig. 5 is a schematic view of a semiconductor 

imager. 

10 Fig. 6 is an electrical schematic diagram of 

circuitry of the imager of Fig. 5. 

Fig. 7A is a representation of the contents of the 
serial register of Fig. 6. 

Fig. 7B is a representation of the contents of the 
15 buffer of Fig. 6. 

Fig. 8 is a electrical schematic diagram of another 

imager . 

Description of the Preferred Embodiments 
20 Fig. 5 shows a CMOS imager 5 0 located on a 

monolithic semiconductor substrate, or chip. The 
illustrated embodiment is constructed to furnish twenty- four 
bit True. Color data, e.g., eight bits representing a red 
color, eight bits representing a green color, and eight bits 
25 representing a blue color, for every photosensitive site 51. 
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Each photosensitive site 51 is a region of the imager 50 
that includes a pixel cell 52. As a result of this 
arrangement, a one-to-one correspondence between pixel cells 
52 of the imager 50 and pixels of a display used to display 
the image captured by the imager 50 is preserved without 
requiring a larger imager, complicated optics, or off-chip 
color interpolation. 

The imager 50 has an on chip color interpolator 58 
which, for each photosensitive site 51, estimates the level 
of the primary colors that are not sensed by the pixel cell 
52 at that photosensitive site 51. The color sensed by the 
pixel cell 52 is determined by a primary color filter of the 
site 51 that covers the pixel cell 52. In this manner, the 
primary color filter (which is a red, green or blue filter) 
covers the cell 52. Each cell 52 senses the level of light 
by measuring the intensity of the light in one of the 
primary color bands (e.g., red, green or blue) but does not 
sense the level of light in the other two primary color 
bands. The interpolator 58 estimates the missing color 
levels for the site 51 by using the outputs of pixel cells 
52 in adjacent photosensitive sites 51 that are sensing 
these color levels. 

The photosensitive sites 51 (and corresponding pixel 
cells 52) are arranged in a rectangular array of rows and 
columns. To estimate the missing color levels for a given 
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photosensitive site 51 (i.e., to estimate the color levels 
not sensed by the site 51) , the interpolator 58 may be 
configured to use pixel cells 52 in the same row, same 
column, or both (e.g., the interpolator 58 may use a block 
5 of pixel cells 52 that surround the given photosensitive 
site 51) . Although may configurations are possible, as 
discussed below, a multi-band column oriented filter pattern 
(See Fig. 2) is assumed, and pixel cells 52 from the same 
row are used in the interpolation. 

10 Fig. 6 shows analog conditioning circuitry 54 to 

perform correlated double sampling of the analog outputs of 
the pixel cells 52 and provide gain control. This circuitry 
receives the analog outputs of the pixel cells 52. The 
circuitry 54 furnishes its output to an analog- to-digital 

15 converter 56 which converts the analog outputs of the pixel 
cells 52 into digital data and supplies the digital data to 
the interpolator 58. After an integration interval has 
passed, the pixel cells 52 have captured a snapshot of the 
image. At that time a column decoder 64 begins routing the 

20 outputs of the pixel cell 52 to the analog conditioning 

circuitry 54 for processing. The decoder 64 sequentially 
selects one row of pixels 52 and serially provides the 
analog outputs of the pixel cells 52 of the row that is 
selected (i.e., provides all of the columns of the selected 

25 row) to the analog conditioning circuitry 54. A control 
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circuit 62 controls the integration of the light by the 
pixel cells 52 and the overall timing of the imager 50. The 
True Color data may be read from the imager 50 at a parallel 
port interface 60. 
5 The interpolator 58 estimates the levels of the 

missing color levels for a given photosensitive site 51 
using the outputs of other p ixel cells 52 that are close to 
the given photosensitive site 51. As one example, the 
interpolator 58 may be configured to use a one dimensional 

10 approach by serially processing photosensitive sites 51 and 
the corresponding pixel cells 52 at the photosensitive sites 
51 from the same row of the array. The processing of a 
given photosensitive site 51 includes retrieving the color 
level sensed by the pixel cell 52 of the given 

15 photosensitive site 51 and estimating the missing color 

levels. The estimation uses the interpolator 58 to form the 
outputs of the last two pixel cells 52 that were processed 
and the next two pixel cells 52 to be processed to estimate 
the two missing color levels for the photosensitive site 51 

20 currently being processed. The interpolator 58 performs a 
weighted average of the outputs from the pixel cells 52 to 
estimate the missing color levels. 

For example, Fig. 7A shows a photosensitive site 51a 
is covered by a blue filter which filters out red and green 

25 light from striking the corresponding pixel cell 52 . To 
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estimate the red light that would otherwise strike the pixel 
cell 52 if not for the blue filter (i.e., to estimate the 
level of red light striking the photosensitive site 51a) , 
the interpolator 58 forms a weighted average of the outputs 
5 of pixel cells 52 in adjacent photosensitive sites 51b and 
51c that are covered by a red filter. Similarly, to 
estimate the green light that would otherwise strike the 
pixel cell 52 if not for the blue filter (i.e., to estimate 
the level of green light striking the photosensitive site 

10 51a) , the interpolator 58 uses a weighted average of the 

outputs of pixel cells 52 in adjacent photosensitive sites 
51d and 51e that are covered by a green filter. 

The estimate of color level for a given 
photosensitive site 51 uses a number of different values. 

15 The weight given by the interpolator 58 to the actual color 
level from another photosensitive site 51 is a function of 
the distance between the given photosensitive site 51 and 
the photosensitive site 51 furnishing the actual color 
level. For example, to estimate the level of green light 

20 striking the photosensitive site 51a (see Fig. 7A) , the 

interpolator 58 might be configured to apply twice as much 
weight to the output of the pixel cell 52 in adjacent 
photosensitive site 51d than to the output of the pixel cell 
52 twice as far away, such as pixel cell 52e. 
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Fig. 6 shows the hardware of the interpolator 58 
including a five stage serial register 66. The least 
significant bits zero to fifteen of the register contain 
eight bit digital representations of actual color levels for 
5 the last two photosensitive sites 51 and corresponding pixel 
cells 52 processed. The most significant bits twenty-four 
to thirty-nine of the register 66 contain eight bit digital 
representations of actual color levels for the next two 
photosensitive sites 51 and corresponding pixel cells 52 to 

10 be processed. The other bits sixteen to twenty- three of the 
register 66 contain an eight bit representation of the 
actual color level for the photosensitive site 51 and 
corresponding pixel cell 52 being processed. 

Each photosensitive site 51 assembles the twenty- 

15 four bit True Color representation in a buffer 74 (of the 

parallel port interface 60) as follows. The interpolator 58 
transfers the bits 16-23 of the register 66 which are 
representative of an actual color level, to the buffer 74 
without any further processing. The interpolator 58 assigns 

20 a weight via scalar multipliers to the values represented by 
the bits 32-39 and 8-15 of the register 66. The 
interpolator 58 also averages (via adders 70 and a "divide- 
by- two" circuitry 72) these values to estimate one of the 
missing color values, and stores the resultant eight bit 

25 color value in the buffer 74. The twenty-four bit 



representation is completed by the interpolator 58 assigning 
a weight to the values represented by the bits 24-31 and 0- 
7, average these values together, and stores the resultant 
eight bit color value in the buffer 75. The twenty-four bit 
5 True Color value may then be retrieved from the buffer 74 
(and from the parallel port interface 60) via an I/O 
interface 76 that is configured to communicate with off chip 
devices . 

Figs. 7A and 7B show the red-green-blue ("RGB") byte 

10 ordering of the stored twenty-four bit color values 69 

circularly rotates, and the most significant byte of the 
color value 69 corresponds to the actual color level sensed 
by the pixel cell 52 in the corresponding photosensitive 
site 51. As an example, for the twenty-four bit color value 

15 69a representative of the color sensed by the pixel cell 52 
in photosensitive site 51a, the most significant byte 
represents the actual blue color level (Bl) sensed by the 
pixel cell in photosensitive site 51a, the next significant 
byte represents the estimated red color level for the 

20 photosensitive site 51a, and the least significant byte 
represents the estimated green color level for the 
photosensitive site 51a. 

The gains of the scalar multipliers 68 (i.e., the 
weighting applied by the interpolator 58) may either be 

25 fixed or programmable. Fig. 6 shows the gains being 



programmable, with the I/O interface 76 having writable and 
readable registers used to program the gains of the 
multipliers . 

The one dimensional color interpolation approach 
5 discussed above can be extended to two dimensional 

interpolation. In such an approach, the outputs from pixel 
cells 52 from more than one row are used to estimate the 
missing color levels of a photosensitive site 51. For 
example, Fig. 8 shows another interpolator 90 of another 

10 imager 100 having three serial, five stage registers 92. 
Similar to the register 66, each register 92 contains 
digital representations of five adjacent pixel cells 52 of 
one of three adjacent different rows. Each register 92 has 
representations from the same column of pixel cells 52. 

15 Thus, the bits of the registers 92 represent the outputs of 
a 5X3 block of pixel cells 52. The interpolator 90 includes 
analog conditioning circuitry 95 and an A/D converter 97 for 
each register 92. The integrator 90 also has a gain circuit 
94 (e.g., scalar multipliers) and an averaging circuit 96 to 

20 provide weighted averaging for the interpolation. The 

imager 100 uses a column decoder 91 that has three serial 
outputs associated with three different adjacent rows of 
pixel cells 52. A control circuit 94 controls the 
integration of the light by the pixel cells 52 and the 

25 overall timing of the imager 100. 
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Other embodiments are within the scope of the 
following claims. For example, other filter patterns, such 
as a checkered filter pattern may cover the array of pixel 
cells. The array may have more pixel cells dedicated to 
5 sensing one of the primary colors than to the other primary 
colors. For example, to improve the perceived luminance of 
the reproduced image, the array may have more pixel cells 
dedicated to sensing green (a color that closely matches the 
luminance of the human eye) color levels. The imager may 
10 represent color in a format other than a True Color 
representation. For example, six bits may be used to 
represent a green color level, five bits may be used to 
represent a blue color level, and five bits may be used to 
represent a red color level. 
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